Large-FOV 3D localization microscopy by spatially variant point spread function generation.

Accurate characterization of the microscopic point spread function (PSF) is crucial for achieving high-performance localization microscopy (LM). Traditionally, LM assumes a spatially invariant PSF to simplify the modeling of the imaging system. However, for large fields of view (FOV) imaging, it becomes important to account for the spatially variant nature of the PSF. Here, we propose an accurate and fast principal components analysis-based field-dependent 3D PSF generator (PPG3D) and localizer for LM. Through simulations and experimental three-dimensional (3D) single-molecule localization microscopy (SMLM), we demonstrate the effectiveness of PPG3D, enabling super-resolution imaging of mitochondria and microtubules with high fidelity over a large FOV. A comparison of PPG3D with a shift-variant PSF generator for 3D LM reveals a threefold improvement in accuracy. Moreover, PPG3D is approximately 100 times faster than existing PSF generators, when used in image plane-based interpolation mode. Given its user-friendliness, we believe that PPG3D holds great potential for widespread application in SMLM and other imaging modalities.

Universal inverse modelling of point spread functions for SMLM localization and microscope characterization.

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single molecule localization, aberration correction and deconvolution. Here we present uiPSF (universal inverse modelling of Point Spread Functions), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single molecule localization microscopy (SMLM). The resulting PSF model enables accurate 3D super-resolution imaging using SMLM. Additionally, uiPSF can be used to characterize and optimize a microscope system by quantifying the aberrations, including field-dependent aberrations, and resolutions. Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system or sample specific characteristics, e.g., the bead size, depth dependent aberrations and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single molecule blinking data.

Design of optimal labeling patterns for optical genome mapping via information theory.

MOTIVATION: Optical genome mapping (OGM) is a technique that extracts partial genomic information from optically imaged and linearized DNA fragments containing fluorescently labeled short sequence patterns. This information can be used for various genomic analyses and applications, such as the detection of structural variations and copy-number variations, epigenomic profiling, and microbial species identification. Currently, the choice of labeled patterns is based on the available biochemical methods and is not necessarily optimized for the application. RESULTS: In this work, we develop a model of OGM based on information theory, which enables the design of optimal labeling patterns for specific applications and target organism genomes. We validated the model through experimental OGM on human DNA and simulations on bacterial DNA. Our model predicts up to 10-fold improved accuracy by optimal choice of labeling patterns, which may guide future development of OGM biochemical labeling methods and significantly improve its accuracy and yield for applications such as epigenomic profiling and cultivation-free pathogen identification in clinical samples. AVAILABILITY AND IMPLEMENTATION: https://github.com/yevgenin/PatternCode.

Neural network-assisted single-molecule localization microscopy with a weak-affinity protein tag.

Single-molecule localization microscopy achieves nanometer spatial resolution by localizing single fluorophores separated in space and time. A major challenge of single-molecule localization microscopy is the long acquisition time, leading to low throughput, as well as to a poor temporal resolution that limits its use to visualize the dynamics of cellular structures in live cells. Another challenge is photobleaching, which reduces information density over time and limits throughput and the available observation time in live-cell applications. To address both challenges, we combine two concepts: first, we integrate the neural network DeepSTORM to predict super-resolution images from high-density imaging data, which increases acquisition speed. Second, we employ a direct protein label, HaloTag7, in combination with exchangeable ligands (xHTLs), for fluorescence labeling. This labeling method bypasses photobleaching by providing a constant signal over time and is compatible with live-cell imaging. The combination of both a neural network and a weak-affinity protein label reduced the acquisition time up to ∼25-fold. Furthermore, we demonstrate live-cell imaging with increased temporal resolution, and capture the dynamics of the endoplasmic reticulum over extended time without signal loss.

Near index matching enables solid diffractive optical element fabrication via additive manufacturing.

Diffractive optical elements (DOEs) have a wide range of applications in optics and photonics, thanks to their capability to perform complex wavefront shaping in a compact form. However, widespread applicability of DOEs is still limited, because existing fabrication methods are cumbersome and expensive. Here, we present a simple and cost-effective fabrication approach for solid, high-performance DOEs. The method is based on conjugating two nearly refractive index-matched solidifiable transparent materials. The index matching allows for extreme scaling up of the elements in the axial dimension, which enables simple fabrication of a template using commercially available 3D printing at tens-of-micrometer resolution. We demonstrated the approach by fabricating and using DOEs serving as microlens arrays, vortex plates, including for highly sensitive applications such as vector beam generation and super-resolution microscopy using MINSTED, and phase-masks for three-dimensional single-molecule localization microscopy. Beyond the advantage of making DOEs widely accessible by drastically simplifying their production, the method also overcomes difficulties faced by existing methods in fabricating highly complex elements, such as high-order vortex plates, and spectrum-encoding phase masks for microscopy.

DBlink: dynamic localization microscopy in super spatiotemporal resolution via deep learning.

Single-molecule localization microscopy (SMLM) has revolutionized biological imaging, improving the spatial resolution of traditional microscopes by an order of magnitude. However, SMLM techniques require long acquisition times, typically a few minutes, to yield a single super-resolved image, because they depend on accumulation of many localizations over thousands of recorded frames. Hence, the capability of SMLM to observe dynamics at high temporal resolution has always been limited. In this work, we present DBlink, a deep-learning-based method for super spatiotemporal resolution reconstruction from SMLM data. The input to DBlink is a recorded video of SMLM data and the output is a super spatiotemporal resolution video reconstruction. We use a convolutional neural network combined with a bidirectional long short-term memory network architecture, designed for capturing long-term dependencies between different input frames. We demonstrate DBlink performance on simulated filaments and mitochondria-like structures, on experimental SMLM data under controlled motion conditions and on live-cell dynamic SMLM. DBlink's spatiotemporal interpolation constitutes an important advance in super-resolution imaging of dynamic processes in live cells.

Fluorescence Microscopy: a statistics-optics perspective.

Fundamental properties of light unavoidably impose features on images collected using fluorescence microscopes. Modeling these features is ever more important in quantitatively interpreting microscopy images collected at scales on par or smaller than light's wavelength. Here we review the optics responsible for generating fluorescent images, fluorophore properties, microscopy modalities leveraging properties of both light and fluorophores, in addition to the necessarily probabilistic modeling tools imposed by the stochastic nature of light and measurement.

DeepOM: single-molecule optical genome mapping via deep learning.

MOTIVATION: Efficient tapping into genomic information from a single microscopic image of an intact DNA molecule is an outstanding challenge and its solution will open new frontiers in molecular diagnostics. Here, a new computational method for optical genome mapping utilizing deep learning is presented, termed DeepOM. Utilization of a convolutional neural network, trained on simulated images of labeled DNA molecules, improves the success rate in the alignment of DNA images to genomic references. RESULTS: The method is evaluated on acquired images of human DNA molecules stretched in nano-channels. The accuracy of the method is benchmarked against state-of-the-art commercial software Bionano Solve. The results show a significant advantage in alignment success rate for molecules shorter than 50 kb. DeepOM improves the yield, sensitivity, and throughput of optical genome mapping experiments in applications of human genomics and microbiology. AVAILABILITY AND IMPLEMENTATION: The source code for the presented method is publicly available at https://github.com/yevgenin/DeepOM.

Roadmap on Deep Learning for Microscopy.

Through digital imaging, microscopy has evolved from primarily being a means for visual observation of life at the micro- and nano-scale, to a quantitative tool with ever-increasing resolution and throughput. Artificial intelligence, deep neural networks, and machine learning are all niche terms describing computational methods that have gained a pivotal role in microscopy-based research over the past decade. This Roadmap is written collectively by prominent researchers and encompasses selected aspects of how machine learning is applied to microscopy image data, with the aim of gaining scientific knowledge by improved image quality, automated detection, segmentation, classification and tracking of objects, and efficient merging of information from multiple imaging modalities. We aim to give the reader an overview of the key developments and an understanding of possibilities and limitations of machine learning for microscopy. It will be of interest to a wide cross-disciplinary audience in the physical sciences and life sciences.

Diffractive optical system design by cascaded propagation.

Modern design of complex optical systems relies heavily on computational tools. These frequently use geometrical optics as well as Fourier optics. Fourier optics is typically used for designing thin diffractive elements, placed in the system's aperture, generating a shift-invariant Point Spread Function (PSF). A major bottleneck in applying Fourier Optics in many cases of interest, e.g. when dealing with multiple, or out-of-aperture elements, comes from numerical complexity. In this work, we propose and implement an efficient and differentiable propagation model based on the Collins integral, which enables the optimization of diffractive optical systems with unprecedented design freedom using backpropagation. We demonstrate the applicability of our method, numerically and experimentally, by engineering shift-variant PSFs via thin plate elements placed in arbitrary planes inside complex imaging systems, performing cascaded optimization of multiple planes, and designing optimal machine-vision systems by deep learning.

Monocular kilometer-scale passive ranging by point-spread function engineering.

Standard imaging systems are designed for 2D representation of objects, while information about the third dimension remains implicit, as imaging-based distance estimation is a difficult challenge. Existing long-range distance estimation technologies mostly rely on active emission of signal, which as a subsystem, constitutes a significant portion of the complexity, size and cost of the active-ranging apparatus. Despite the appeal of alleviating the requirement for signal-emission, passive distance estimation methods are essentially nonexistent for ranges greater than a few hundreds of meters. Here, we present monocular long-range, telescope-based passive ranging, realized by integration of point-spread-function engineering into a telescope, extending the scale of point-spread-function engineering-based ranging to distances where it has never been tested before. We provide experimental demonstrations of the optical system in a variety of challenging imaging scenarios, including adversarial weather conditions, dynamic targets and scenes of diversified textures, at distances extending beyond 1.7 km. We conclude with brief quantification of the effect of atmospheric turbulence on estimation precision, which becomes a significant error source in long-range optical imaging.

THY1-mediated mechanisms converge to drive YAP activation in skin homeostasis and repair.

Anchored cells of the basal epidermis constantly undergo proliferation in an overcrowded environment. An important regulator of epidermal proliferation is YAP, which can be controlled by both cell-matrix and cell-cell interactions. Here, we report that THY1, a GPI-anchored protein, inhibits epidermal YAP activity through converging molecular mechanisms. THY1 deficiency leads to increased adhesion by activating the integrin-ß1-SRC module. Notably, regardless of high cellular densities, the absence of THY1 leads to the dissociation of an adherens junction complex that enables the release and translocation of YAP. Due to increased YAP-dependent proliferation, Thy1-/- mice display enhanced wound repair and hair follicle regeneration. Taken together, our work reveals THY1 as a crucial regulator of cell-matrix and cell-cell interactions that controls YAP activity in skin homeostasis and regeneration.

Quantifying cell-cycle-dependent chromatin dynamics during interphase by live 3D tracking.

The study of cell cycle progression and regulation is important to our understanding of fundamental biophysics, aging, and disease mechanisms. Local chromatin movements are generally considered to be constrained and relatively consistent during all interphase stages, although recent advances in our understanding of genome organization challenge this claim. Here, we use high spatiotemporal resolution, 4D (x, y, z and time) localization microscopy by point-spread-function (PSF) engineering and deep learning-based image analysis, for live imaging of mouse embryonic fibroblast (MEF 3T3) and MEF 3T3 double Lamin A Knockout (LmnaKO) cell lines, to characterize telomere diffusion during the interphase. We detected varying constraint levels imposed on chromatin, which are prominently decreased during G0/G1. Our 4D measurements of telomere diffusion offer an effective method to investigate chromatin dynamics and reveal cell-cycle-dependent motion constraints, which may be caused by various cellular processes.

Synthetic cells with self-activating optogenetic proteins communicate with natural cells.

Development of regulated cellular processes and signaling methods in synthetic cells is essential for their integration with living materials. Light is an attractive tool to achieve this, but the limited penetration depth into tissue of visible light restricts its usability for in-vivo applications. Here, we describe the design and implementation of bioluminescent intercellular and intracellular signaling mechanisms in synthetic cells, dismissing the need for an external light source. First, we engineer light generating SCs with an optimized lipid membrane and internal composition, to maximize luciferase expression levels and enable high-intensity emission. Next, we show these cells' capacity to trigger bioprocesses in natural cells by initiating asexual sporulation of dark-grown mycelial cells of the fungus Trichoderma atroviride. Finally, we demonstrate regulated transcription and membrane recruitment in synthetic cells using bioluminescent intracellular signaling with self-activating fusion proteins. These functionalities pave the way for deploying synthetic cells as embeddable microscale light sources that are capable of controlling engineered processes inside tissues.

High-Throughput Imaging of CRISPR- and Recombinant Adeno-Associated Virus-Induced DNA Damage Response in Human Hematopoietic Stem and Progenitor Cells.

CRISPR-Cas technology has revolutionized gene editing, but concerns remain due to its propensity for off-target interactions. This, combined with genotoxicity related to both CRISPR-Cas9-induced double-strand breaks and transgene delivery, poses a significant liability for clinical genome-editing applications. Current best practice is to optimize genome-editing parameters in preclinical studies. However, quantitative tools that measure off-target interactions and genotoxicity are costly and time-consuming, limiting the practicality of screening large numbers of potential genome-editing reagents and conditions. Here, we show that flow-based imaging facilitates DNA damage characterization of hundreds of human hematopoietic stem and progenitor cells per minute after treatment with CRISPR-Cas9 and recombinant adeno-associated virus serotype 6. With our web-based platform that leverages deep learning for image analysis, we find that greater DNA damage response is observed for guide RNAs with higher genome-editing activity, differentiating even single on-target guide RNAs with different levels of off-target interactions. This work simplifies the characterization and screening process of genome-editing parameters toward enabling safer and more effective gene-therapy applications.

Deep-ROCS: from speckle patterns to superior-resolved images by deep learning in rotating coherent scattering microscopy.

Rotating coherent scattering (ROCS) microscopy is a label-free imaging technique that overcomes the optical diffraction limit by adding up the scattered laser light from a sample obliquely illuminated from different angles. Although ROCS imaging achieves 150 nm spatial and 10 ms temporal resolution, simply summing different speckle patterns may cause loss of sample information. In this paper we present Deep-ROCS, a neural network-based technique that generates a superior-resolved image by efficient numerical combination of a set of differently illuminated images. We show that Deep-ROCS can reconstruct super-resolved images more accurately than conventional ROCS microscopy, retrieving high-frequency information from a small number (6) of speckle images. We demonstrate the performance of Deep-ROCS experimentally on 200 nm beads and by computer simulations, where we show its potential for even more complex structures such as a filament network.

Multiplexed PSF Engineering for Three-Dimensional Multicolor Particle Tracking.

Three-dimensional spatiotemporal tracking of microscopic particles in multiple colors is a challenging optical imaging task. Existing approaches require a trade-off between photon efficiency, field of view, mechanical complexity, spectral specificity, and speed. Here, we introduce multiplexed point-spread-function engineering that achieves photon-efficient, 3D multicolor particle tracking over a large field of view. This is accomplished by first chromatically splitting the emission path of a microscope to different channels, engineering the point-spread function of each, and then recombining them onto the same region of the camera. We demonstrate our technique for simultaneously tracking five types of emitters in vitro as well as colocalization of DNA loci in live yeast cells.

3D printable diffractive optical elements by liquid immersion.

Diffractive optical elements (DOEs) are used to shape the wavefront of incident light. This can be used to generate practically any pattern of interest, albeit with varying efficiency. A fundamental challenge associated with DOEs comes from the nanoscale-precision requirements for their fabrication. Here we demonstrate a method to controllably scale up the relevant feature dimensions of a device from tens-of-nanometers to tens-of-microns by immersing the DOEs in a near-index-matched solution. This makes it possible to utilize modern 3D-printing technologies for fabrication, thereby significantly simplifying the production of DOEs and decreasing costs by orders of magnitude, without hindering performance. We demonstrate the tunability of our design for varying experimental conditions, and the suitability of this approach to ultrasensitive applications by localizing the 3D positions of single molecules in cells using our microscale fabricated optical element to modify the point-spread-function (PSF) of a microscope.

Automated Analysis of Fluorescence Kinetics in Single-Molecule Localization Microscopy Data Reveals Protein Stoichiometry.

Understanding the function of protein complexes requires information on their molecular organization, specifically, their oligomerization level. Optical super-resolution microscopy can localize single protein complexes in cells with high precision, however, the quantification of their oligomerization level, remains a challenge. Here, we present a Quantitative Algorithm for Fluorescent Kinetics Analysis (QAFKA), that serves as a fully automated workflow for quantitative analysis of single-molecule localization microscopy (SMLM) data by extracting fluorophore "blinking" events. QAFKA includes an automated localization algorithm, the extraction of emission features per localization cluster, and a deep neural network-based estimator that reports the ratios of cluster types within the population. We demonstrate molecular quantification of protein monomers and dimers on simulated and experimental SMLM data. We further demonstrate that QAFKA accurately reports quantitative information on the monomer/dimer equilibrium of membrane receptors in single immobilized cells, opening the door to single-cell single-protein analysis.

Learning Optimal Wavefront Shaping for Multi-Channel Imaging.

Fast acquisition of depth information is crucial for accurate 3D tracking of moving objects. Snapshot depth sensing can be achieved by wavefront coding, in which the point-spread function (PSF) is engineered to vary distinctively with scene depth by altering the detection optics. In low-light applications, such as 3D localization microscopy, the prevailing approach is to condense signal photons into a single imaging channel with phase-only wavefront modulation to achieve a high pixel-wise signal to noise ratio. Here we show that this paradigm is generally suboptimal and can be significantly improved upon by employing multi-channel wavefront coding, even in low-light applications. We demonstrate our multi-channel optimization scheme on 3D localization microscopy in densely labelled live cells where detectability is limited by overlap of modulated PSFs. At extreme densities, we show that a split-signal system, with end-to-end learned phase masks, doubles the detection rate and reaches improved precision compared to the current state-of-the-art, single-channel design. We implement our method using a bifurcated optical system, experimentally validating our approach by snapshot volumetric imaging and 3D tracking of fluorescently labelled subcellular elements in dense environments.